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Abstract 



A variety of computer test applications have been developed in recent ye;irs and computer-based testing has 
proved feasible in terms of the potential for adequate degrees of reliability and validity. A more difficult proposition 
is the direct, concurrent equivalence of computer and paper versions of a given examination. Many instances have 
been noted of score differences across the two test modes, commonly termed mode effect. The purpose of this 
resesirch was to determine whether examinee characteristics are systematically related to mode effect across paper 
and computer versions of the same examinations. 

The data for this study were obtained from the Educational Testing Service's Graduate Record Examination 
(GRE) Program. The GRE Program conducted a Computer-Based Testing Pilot study in the Fall of 1991; secondary 
analyses of the results of that pilot study were conducted for purposes of this research. 

The examinee ch. racteristics that were investigated as possible contributors to mode effect included: 
(1) demographic variables (gender, racial/ethnic background, and age), (2) computer use variables (variety of 
computer experience, frequency of computer use, frequency of mouse use, test mode preference), and (3) test taking 
strategy variables (test strategy preference, tendency to omit items, tendency to review items). 

The data were analyzed through two methods. First, the typical method found in the literature on 
comparability studies was employed. In this method, performance across the paper and computer version is treated 
as a continuous, dependent variable. Secondly, because a mode effect in a small subset of examinees could be 
masked when all subjects are considered together, a method which isolated those examinees most affected by test 
mode was also used. For this method, mode effect was treated as a three-level, categorical, independent variable. 

The data in this study demonstrated mode effect and supported the conception of a small subset of 
examinees whose performance was more affected by test mode than was that of the total sample of examinees. The 
search for examinee characteristics that explain the occurrence of mode effect, however, yielded inconsistent results. 
The variables investigated in this study showed only relatively weak relationships to mode effect. 
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Introduction 

When computer-based tests are developed from pre-existing paper and pencil tests information is usually 
required concerning the comparability of examination scores obtained in one mode to scores obtained in the other 
mode. Bunderson, lnouye, & Olsen, (1989) state that, for the field of computer-based testing, "The fundamental 
research question...is the equivalence of scores between a computerized version of a test and the original version" 
(p. 378). 

Comparability of scores obtained on computer and paper versions of an examination is needed to 
establish validity across forms and to ensure that norms which were developed from a paper-administered version 
are useable for the computer version. The comparability of test scores is also a vital issue because of the need 
to provide fair testing for all examinees. According to the Joint Committee's Standards for Educational and 
Psychological Testing , when alternate forms of an examination are in use, "it should be a matter of indifference 
to anyone taking the test or to anyone using the test results" which form of the examination was administered 
(Committee to Develop Standards for Educational and Psychological Testing [Joint Committee], 1985, p. 31). 
When test mode affects performance, this is clearly not the case. 

Mazzeo and Harvey (1988) conducted an extensive review of comparability studies, summarizing 
research on computerized personality, ability, and achievement tests. After finding that mean score differences 
across mode frequently occurred, their conclusion was that "a critical issuc.is the equivalence of computer- 
generated scores to paper-and-pencil scores" (p. 2). 

Although comparability can be approached from the total test score perspective, or from the item level 
perspective, this study analyzed comp;irabiliiy from the examinee mode effect perspective. Some researchers 
suggest that small overall differences in performance across test modes may actually be the result of more 
sizable differences affecting only a small number of examinees (Wise, Barnes, Harvey, & Plake, 1989; Wise & 
Plake, 1989). This comparability study was designed to look at subgroups of examinees in order to isolate the 
scores of those who appear to be most affected by test mode, and then to attempt to find the examinee 
characteristics which distinguish those examinees who are affected by test mode from those who are not. 
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Examinee Characteristics as Contributors to Mode Effect 

Research on the relationship between various examinee characteristics and mode effect has not been 
conclusive. Gender, race or ethnic background, and age are ;unong the demographic attributes which have been 
investigated (Buhr & Legg, 1989; Johnson & Mihal, 1973; Johnson & White, 1980; Llabre & Froman, 1987; 
Moe & Johnson, 1988; Sorensen, 1985). Although decisive evidence of die relationship between these 
chiiracteristics ;ind mode effect has not been obtained, grounds for concern can be found in the results of national 
surveys on equity of computer access (Becker & Sterling, 1987; McPhail, 1985). Across elementary and 
secondary levels, schools with predominantly minority students were more likely to have no computers or to 
have higher student-per-computer ratios. Within schools, males more than females tended to take part in 
optional computer courses and computer activities. To the extent that certain subgroups have unequal access to 
computers, concerns about the consequences of amount of computer experience on performance under computer- 
administered test conditions become applicable. 

Several aspects of examinee interactions with computers have been investigated as potential sources of 
mode effect; thus far, clear-cut results have not been obtained (Eaves & Smith, 1986; Lee, 1986). A pattern of 
lower scores for examinees with lees computer experience is frequently seen, although the score differences are 
often not statistically significant (Wise et al., 1989). Several studies have investigated the relationship between 
examinee performance on computer tests and a stated preference for paper or computer versions of a test, but no 
direct relationship has been consistently found (Bugbee & Bernt, 1990; Buhr & Legg, 1989; Koch & Patience, 
1978). 

Another area of research is that of the relationship between test taking strategy preferences and 
computer test scopes (Rocklin & O'Donnell, 1987; Wise & Plake, 1989). The operation of taking a test on 
computer is different in certain procedural ways; whether all examinees are able to adapt to those differences 
without impact on their performance is not certain. Specifically, the interaction of examinees' test-taking 
strategies with test flexibility (i.e., the presence or absence of features which enable a computer test-taker to omit 
jind revise answers) appears to be important (Green, 1991; Spray, Ackennan, Reckase, & Carlson, 1989; Sachar 
& Fletcher, 1978; Ward, Hooper, & Hannafin, 1989). Wise and Plake (1990) have stated 



...individual examinees vary greatly in their item skipping, answer reviewing, and answer changing 
behavior; yet to be ascertained is whether the scores of examinees who strongly prefer to return to items 
would be affected by denying them that opportunity (p. 8). 

Methods 

This study w;is a secondary ;inalysis of data from the Educational Testing Service's (ETS) pilot study of 
a computer version of the Graduate Record Examination (GRE) General Test. During the standard, paper 
administration of the GRE conducted in October, 1991, examinees who took a specific test form were given the 
opportunity to enlist in the pilot study. Examinees were offered a $50 honorarium for participating and were 
given the option, if the pilot study proved successful, to have their scores from the computer administration of 
the examination added to their score records. Since subjects were thus offered an opportunity to better their 
scores in a high-stakes test environment, motivated pjuticipation was encouraged. 

Examinees from the October administration who agreed to participate were then re-tested on an alternate 
test form during November or December of 1991. A small number of examinees were given this second test in 
paper form, to provide ETS with cross-checks in the research design. For the majority of examinees, however, 
the second test was administered on computer. Only those examinees who took both a paper and a computer 
version are considered in this secondary analysis. The total number of examinees whose scores were used in the 
secondary analysis was 1,1 14. This sample of examinees is typical of GRE test takers in terms of test scores, 
gender, and ethnicity (ETS, 1992). 
Instruments 

The standard, paper-version of the GRE General Test consists of two sections each of Quantitative, 
Verbal, and Analytical scales. One non-operational section is typically also administered in order to protest new 
items; scores on this seventh section were not used in this study. KR-20 estimates of reliability for the test form 
used in the pilot study yielded a .91 estimate of reliability for the Verbid scale, .92 for the Quantitative scale, and 
.88 for the Analytical scale (ETS, 1992). 

The computer test was completely parallel to the paper test in terms of standard sections, item types, 
and numbers of items. The seventh section in this version w;ts a non-flexible Verbal section and will be 
discussed in further detail later. Because the computer took two to three seconds to access and present each 



item* the time allotment for each section on the computer version was 32 minutes, rather than 30 minutes. The 
computer test was administered on IBM PCs with VGA monitors and ran under Microsoft Windows. 
Administration of the computer test was immediately preceded by a computer tutorial on the use of the test 
administration software. KR-20 estimates on this test form resulted in .92 for both the Verbal and Quiintitative 
scales and .89 for the Analytical scale. (ETS, 1992). 

Demographic information on examinees was obUiined through a questionnaire completed by examinees 
as part of the registration process. Information about examinees' prior computer experience and their reactions 
to the computer version were collected through a computer test survey, administered to examinees after they had 
completed the computer test. 
Residual Difference Scores and Methods of Analysis 

The presence of a mode effect was determined by evaluating the difference in examinee perform;incc 
across computer and paper versions of the parallel test forms. For the most part, these differences were 
examined in terms of residual scores rather than difference scores, because of the tendency of simple difference 
scores to be unreliable (Linn, 1988). The residual difference scores were calculated as the difference between 
the examinees* predicted computer test scores (based on a linear regression of the computer version scores on the 
paper version scores) and their actual computer test scores. Thus, a negative residual difference score would 
suggest that an examinee performed less well on the computer version than his or her own paper version score 
would predict. The residual scores were computed separately for each test scale. 

Two methods were used in this study to investigate the occurrence of test mode effect. The first 
method (Method 1) followed the more typical process of assessing the presence of mode effect through 
comparisons of examinee performance across the two modes (computer and paper versions). For this method 
examinees' residualized difference cores on the Verbal, Quantitative, and Analytical scales were used as 
continuous, dependent variables. 

An alternative method was also used (Method 2), in order to investigate the hypothesis that mode effect 
occurs predominantly in a small subset of examinees (Wise et al., 1989). For this method, the approach was to 
isolate the scores of those subjects whose performance appu. *o be most affected by mode of test 
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administration. Based on their residual difference scores, subjects were therefore classified according to three 
levels of mode effect (mode effect favoring the paper test score, no mode effect, and mode effect favoring the 
computer test score) using plus or minus one standiird deviation ;is the criteria. Examinees whose resided scores 
were within one standard deviation of zero were classified its having no mode effect; those with residuals one 
standard deviation above the me;in were assigned to the computer mode effect group; and those with residual 
difference scores one standiird deviation beiow the mean were assigned to the paper mode effect group. One 
standard deviation was set as the cut-off in order to provide for sufficient sample size in the two extreme mode 
effect groups; since the residual difference scores were approximately normally distributed, roughly two-thirds of 
the sample were assigned to the no mode effect group. These mode effect groups were obtained separately for 
the Verbal, Quantitative, and Analytical scales. 

It is important to realize that the fact that extreme residuals were found under Method 2 is not at issue; 
the presence of extreme values is a natural occurrence when residual scores are computed. Instead, the point of 
interest under Method 2 is the relationship between the mode effect groupings and examinee characteristics. 

Results 

Omnibus Test for Mode Effect 

The first analysis conducted was to determine whether an overall mode effect was present in the data. 
Hotclling's T 2 was performed on the difference between scale means across test mode. Simple difference scores 
(computer version mean score minus paper version mean score) on the three scales were used as the dependent 
measures in this analysis. A significant result was found (T 2 = 1,728.36, £< .01). Follow-up univariate t-tests 
on each examination scale were also found significant: Verbid, t (1 1 12) = -2.22, j> < .05; Quantitative, t (1 1 12) = 
-40.40, £< .01; and Analytical, t (1112) = -11.76, £< .01. 

An examination of test means (Table 1) indicates that, across all three test scales, examinees performed 
significantly better on the computer version of the examination than on the paper version of the same lest. 
However, the fact that test mode and test order were perfecUy confounded in the origimd design of the computer 
test pilot study makes interpretation of these omnibus results difficult. The overall improvement in scores on the 
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computer test could have resulted from practice effect 1 ' rather than test mode differences. This is the conclusion 
which ETS has made about the test, based in pmt on additional data collected during the pilot study which were 
not used in this secondary amdysis (ETS, 1992), However, even if retest effects caused the ovendl 
improvement in scores on the computer version of the examination, the kirge residual difference scores obtained 
by some examinees is not addressed. Spray et ;d. (1989) have suggested that item level differences across test 
mode may ancel each other out on a total test score; in the same way, positive and negative differences in 
examinee performance across test mode may cancel each other out at the toud group level. The differences 
between mean scores across levels of the mode effect group variable (Table 2) suggest that on each examination 
scale, a subset of examinees performed very differently across test mode. The remainder of analyses conducted 
were aimed at determining examinee cluiracteristics which define those examinees. 



Table 1 

Means and Standard Deviations of Test Scores by Scale and Test Mode 



Scale 


No. of 
Items 


Test Mode 
Paper 


Computer 


Verbal 


76 






M 




49.93 


50.31 


SD 




12.42 


12.05 


Quantitative 


CO 






M 




36.44 


42.02 


SD 




10.81 


10.59 


Analytical 


50 






M 




32.21 


33.96 


SD 




8.49 


8.40 



Note. N = 1 1 14. 
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Tabic 2 

Mean Scores and Standard Deviations by Tcsl Mode. Mode Effect Group, and Scale 



Mode Effect Group 



Test Mode 



Computer 



No Effect 



Paper 



Computer 

M 

SD 

Paper 
M 
SD 



Verbal (n of items = 76) 
n = 159 n = 788 



56.50 
9.53 



47.92 
11.64 



51.31 
11.62 



50.87 
12.89 



n = 167 



39.71 
9.62 



47.39 
10.17 



Computer 

M 
SD 

Paper 

M 

SD 



Quantitative (n of items = 60) 
n = 105 n = 907 



46.55 
6.89 



32.55 
7.88 



42.82 
10.27 



37.35 
11.09 



n = 101 



30.30 
8.77 



32.36 
8.99 



Computer 
M 
SD 

Paper 
M 
SD 



Analytical (n of items = 50) 
n = 94 n = 915 



39.61 
5.55 



29.03 
6.91 



34.63 
7.5 



32.83 
8.40 



n = 105 



23.04 
8.41 



29.62 
9.54 
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Demographic Variables 

The next analysis conducted was an extensive elimination of the ulationship between mode effect and 
examinee demographic characteristics of gender, racial/ethnic background, and age. Sixty-one percent of the 
examinees were female, and 39% were male. The viable age was a dichotomy, with W/< of the examinees 
categorized as "Older" (those 30 ye;trs of age and older) and 82% categorized as "Younger", Table 3 provides the 
frequencies of examinees by race. 



Table 3 

Frequency Distribution of Examinee Sample by Race 



Race 


Frequency 


Percent 


Asians 


74 


6.8 


Blacks 


87 


8.0 


Hispanics 


50 


4.6 


Whites 


849 


77.6 


Other 


34 


3.1 



Note. N = 1094. 



Method 1 . To determine if there were test mode effects due to examinee chiiracteristics, data were subjected 
to a three-factor MANOVA (Table 4). Significant main effects were found for gender, and for race; significant 
interaction effects were found for gender by race, and for race by age. The three-way gender by race by age 
interaction was not significant. 
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Tabic 4 

MANOVA of Residual Difference Scores on Verbal, Quantitative, and Analytical Scales by Gender, Race, and Auc 





UI 


w liKs Lainnoa 


17 

r 


Gender 


3, 1067 


.yy 


4.33* 


Race 


12. 2823 


.97 


2.92* 


Age 


3, 1067 


.yy 


2.45 


Gender * Race 


12, 2823 


.97 


2.57* 


Gender * Age 


3, 1067 


1.00 


1.24 


Race * Age 


12, 2823 


.97 


2.95* 


Gender * Race * Age 


12, 2823 


.99 


1.24 



*£< .01. 



Follow-up three-factor ANOVAs were conducted for each of the three test scales. Table 5 presents the 
ANOVA on the Verbal residual difference scores. Significant main effects were found for gender ;ind race; 
significant interactions were found for gender by race, and race by age. Dunn's multiple comparison test w;u used 
as a follow-up test to oach of the significant interaction effects, as relatively few pairwise comparisons were planned. 

A visual examination of Figure 1 (a plot of the interaction effect between race ;ind gender) suggests that 
at some levels of race, males' residual difference scores on the Verbal scale were negatively impacted compiled to 
females. However, no individual pairwise comparisons were found to be significant for the gender by race 
interactions. The Asian, Black, and Hispanic levels of race were collapsed and categorized as "non-White" in order 
to analyze the effect across that group. (Hie category "Other" was disregarded at this point, since additional data 
about the ethnic make-up of examinees in that level were unavailable). When the comparison for the White versus 
non-White level of race was conducted, a significant difference was found between mean scores for males and 
females (Dunn's = 3.59, £< .01). This suggests that non-White males' performance on the computer version of the 
test was less than their performance on the paper version predicted, as compared to non-White females. The 
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significant main effects for gender and race could not be interpreted directly because of the significant interaction 
effects. 



Table 5 

Analysis of Variance of Residue Difference Scores by Gender, Race, and Age on Verbal Scale 



Source 


df 


MS 


F 


Gender (G) 


1 


135.24 


4.99* 


Race (R) 


4 


113.57 


4.19** 


Age (A) 


1 


102.96 


3.80 


GR 


4 


85.42 


3.15* 


GA 


1 


61.38 


2.26 


RA 


4 


152.69 


5.63** 


GRA 


4 


54.95 


2.03 


Error (S/GRA) 


1069 


27.12 




Total 


1088 







* £ < .05. ** J2< .01. 



A plot of the interaction effect between race and age is shown in Figure 2. A significant difference was 
found between means for older and younger Blacks (Dunn's = 3.50, £ < .01). This suggests that older Blacks 
performed less well on the computer version of the examination than their paper version scores predicted, as 
compared to younger Blacks. (Very low n's in the Asian, Hispanic, and Other levels of race by age may have 
reduced power for these post-hoc analyses.) 
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Female 
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Figure 1 ; Interaction of Race and Gender on Verbal Residual Difference Scores. 
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The three factor ANOVA on the Quantitative residual difference scores (Table 6) revealed a significant main 
effect for gender and a significant interaction effect between race mid age. A plot of this interaction effect is shown 
in Figure 3. Again, the only pairvvise comparison that was significant at the .05 hypothesiswise level was for Blacks 
(Dunn's = 3.34, XL < .01). The same pattern held here its in the Verbal scale: older Blacks had a significantly lower 
mean residual difference score than did younger Blacks. Relative to the significant mmn effect for gender, males 
had a significantly higher mean residual difference score on the Quantitative scale than did females. This would 
suggest that, on the Qualitative scale, the overall performance of males on the computer version was higher than 
would be expected based on their paper version scores. For females, the average performance on the computer 
version was less than would be expected, based on their own paper version performance. 



Table 6 

Analysis of Variance of Residual Difference Scores by Gender, Race, and Age on Quiintitative Scale 



Source 


df 


MS 


F 


Gender (G) 


1 


128.05 


6.70** 


Race (R) 


4 


28.13 


1.47 


Age (A) 


1 


31.26 


1.64 


GR 


4 


37.80 


1.98 


GA 


1 


9.03 


0.47 


RA 


4 


51.15 


2.68* 


GRA 


4 


12.50 


0.65 


Error (S/GRA) 


1069 


19.11 




Total 


1088 







*£< .05. ** £ < .01. 




The three factor ANOVA of gender, racial/ethnic background, and age on Analytical residual difference 
scores indicated a significant main effect for race (Table 7); no other significant effects were observed. wSince all 
possible pairwise comparisons of the race variable were to be performed, Tukey's multiple comparison test was 
selected. Tukey's post hoc test on the main effect for race revejiled a significant difference between Blacks and 
Asians (2 < .05). Asians had a significantly greater mean residual score in the Analytical scale than did Blacks, 
indicating that Asians performed better on the computer version than their paper version scores would predict, as 
compjired to Blacks. 
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Table 7 

Analysis of Variance of Residual Difference Scores hv Gender. Race, and Age on Analytical Scale 



Source 


df 


MS 


F 


Gender (G) 


1 


0.19 


0.01 


Race (R) 


4 


94.03 


4.46* 


Age (A) 


1 


71.69 


3.40 


GR 


4 


46.48 


2.21 


GA 


1 


19.66 


0.93 


RA 


4 


42.90 


2.04 


GRA 


4 


17.76 


0.84 


Error (S/GRA) 


1069 


21.06 




Total 


1088 







* n < .01. 



Method 2 . Under the Method 2 approach in this study, residual difference scores were used to categorize 
examinees according to level of mode effect. In order to investigate the possibility that mode effect groups were 
defined by crossed demographic variables (e.g. younger Hispanic females), log-linear analyses were conducted. The 
log-linear analyses performed were asymmetrical in nature; that is to say, only those associations affecting the mode 
effect group were of interest. The demographic variables of gender, racial/ethnic background, and age group served 
as the explanatory variables in these analyses, while mode effect group was the categorical respondent variable. 

In log-linear analysis, the initial analysis provides a screening of four families of models. First, the null 
model (Model 0) is tested, in which the marginal frequencies of each variable are used to establish expected 
frequencies in each cell. Second, the main effects only model is tested (Model 1), in which the independent effects 
of the three explanatory variables contribute to the expected frequencies. Third, the main effects and two-way 
interaction model is tested (Model 2), in which the three two-way interaction terms (gender X race, gender X age. 
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nice X age) are included in addition to the imiin effects. Finally, the saturated model is tested (Model 3), in which 
the three-way interaction (gender X race X age) is added to the previous model (Kennedy, 1983). 

The tests for the screening of these four ftunilies of models with mode effect groups iire presented for the 
Verbid scale in Table 8. The goodness of fit chi-squiire statistics provided tests of the amount of discrepancy 
between the observed cell frequencies and the expected frequences provided by each model. A hirge value of chi- 
square (i.e., one with a probability of less than .05 under a true null hypothesis), indicates that the observed cell 
frequencies are sufficiently different from those expected that the model is not tenable. The goodness of fit tests 
in Table 8 indicated that any of the models provide an adequate fit for the data. 

in addition to die goodness of fit test, the likelihood ratio chi-square is used to test the difference in fit 
between the reduced model relative to the model with additional effects included. This test, referred to as a 
component analysis, is analogous to the more familiar test for a change in R 2 in multiple regression analyses. The 
component analyses in Table 8 indicated that none of the differences in fit between successive pairs of models 
proved to be statistically significant. This indicates that the variables, as measured here, are not strongly related to 
mode effect group. 

Table 8 

Log-linear Analysis of Mode Effect Group by Gender, Racial/Ethnic Background and Age for Verbal Scale 



Goodness of Fit Component 

Model L 2 df £ L 2 df 



0 


44.50 


38 


0.22 








1 


31.18 


26 


0.22 


13.32 


12 


0.35 


2 


2.37 


8 


0.97 


28.81 


18 


0.06 


3 




0 




2.37 


8 


0.97 



18 



16 

Table 9 presents the log-linear analysis conducted on the Quantitative scale. The goodness of fit tests 
presented here indicated that the null model (Model 0) did not provide ;tn acceptable fit to the observed data 
(L 2 = 55.44, £ < .03), but that any of the other three models fit the data adequately. The component juialyses 
indicated that the difference in fit between the null model and the main effects model (Model 1) is statistically 
significant, L : = 33.59, £ < .01. The addition of two-way interactions (Model 2) to the main effects model did 
not significantly enhance the degree of fit, nor did the addition of the three-way interaction to the two-way 
interaction model. 

The results of the goodness of fit and component analyses suggested that at lejust one of the explanatory 
variables (gender, racial/ethnic background, and age group) w;is associated with mode effect, but that no 
significant interactions between these explanatory variables were evident. To determine which explanatory 
variables were related to mode effect, additional analyses were conducted. Because the three explanatory 
variables were correlated with each other, these analyses were conducted by testing the fit of models in which 
only two of these explanatory variables were included. Tests of the difference in the fit of each of these models, 
relative to the model in which all main effects were included, provide a test of the significance of each 
explanatory variable alter adjusting for the other explanatory variables. As indicated by the component analyses 
in Table 10, significant effects were obtained for the explanatory variables of gender and age. The variable of 
examinee race, however, showed no significant relationship with mode effect after adjusting for gender and age. 

The bivariate frequency distribution of gender on mode effect for the Quantitative scale indicates that 
female examinees, relative to males, were more frequently found in the no mode effect group (59%) and the 
paper mode effect group (83%) compared to the computer mode effect group (56%). Similarly, the bivariate 
distribution of examinee age and mode effect group for the Quantitative scale indicates that younger examinees, 
relative to older examinees, were more frequently found in the no mode effect group (82%) and the computer 
mode effect group (84%), compared to the paper mode effect group (74%). 
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Table 9 

Lou-lino*"" Analysis of' Mode Effect Group hy Gender, Racial/Ethnic Background and A^ie lor Quantitative Scale 



Goodness of Fit Component 



Model 


L : 


df 


J! 




df 




0 


55.44 


38 


0.03 








1 


21.85 


26 


0.70 


33.59 


12 


0.01 


2 


1.33 


8 


1.00 


20.52 


18 


0.30 


3 




0 




1,33 


8 


0.99 



Table 10 

Component Analysis on L 2 -Ch;tnge for Main Effects of Gender, Racial/Ethnic Background and Ace on Mode 
Effect for Qualitative Scale 





Goodness of Fit 






Component 






Model V 


df £ 


L 2 


df 




All main effects 


21.85 


26 


0.70 








Race & Age only 


1.67 


28 


1.00 


20.18 


2 


0.01 


Race & Gender only 


15.32 


28 


0.91 


6.53 


2 


0.05 


Age & Gender only 


16.74 


34 


0.54 


5.11 


8 


0.70 



The results of the log-linear analysis on the Analytical scale are given in Table 11. The results of the 
goodness of fit tests indicated that any of the models provide ;in adequate fit for the data. However, results of 
the component analyses indicated that the difference in fit between the null model and the main effects model 
(Model 1) was statistically significant, L~ = 25.16, p_ < .05. Neither the addition of two-way interactions (Model 
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2) lo the main effects model, nor the addition of the three-way interaction (Model 3) to the two-way interaction 
model significantly enfcinced the fit to the ohserved data. These results suggested that at least one of the 
explanatory variables (gender, racial/ ethnic background, find age group) was jissociated with mode effect, but 
that no significant interactions between these expUmatory variables was evident. 

Follow-up mialyses were performed to determine which explanatory v;uiables were related to mode 
effect. The component amdyses in Table 12 indicate a signifiaint relationship between age and mode effect. 
The variables of gender and race both failed to show a significant relationship with mode effect after adjusting 
for the other variables. 

Examination of the biv;iriate distribution of age on mode effect for the Analytical scale indicates that 
younger examinees, relative to older examinees, were more frequently found in the no mode effect group (84%) 
and the computer mode effect group (83%), compared to the paper mode effect group (70%). 

Table 11 

Log-linear Analysis of Mode Effect Group by Gender, Racial/Ethnic Background and Age for Analytical Scale 



Goodness of Fit Component 

Model L 2 df n L 2 df 



0 


37.13 


38 


0.51 








1 


11.97 


26 


0.99 


25.16 


12 


0.02 


2 


1.28 


8 


1.00 


10.69 


18 


0.90 


3 




0 




1.28 


8 


0.99 
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Tabic 12 

Component Analysis on L : -Chancc for Main Effects of Gender, Racial/Ethnic Background and Age on Mode 
Effect for Analytical Scale 



Goodness of Fit Component 

Model L 2 df £ < L : df 



All main effects 


12.04 


26 


0.99 








Race & Age only 


12.91 


28 


0.99 


.87 


2 


0.99 


Race & Gender only 


19.73 


28 


0.87 


7.69 


2 


0.02 


Age & Gender only 


26.03 


34 


0.83 


13.99 


8 


0.09 



Computer Experience 

The next analysis performed was an investigation into the Jlationship between mode effect and variety 
and amount of computer experience. Overall, this sample of examinees was highly computer literate. Only 1% 
of Hie examinees in this study had never used a computer before taking the computer test, and only 15% had 
never used a mouse previously. 

Three questions from the computer test survey were used to measure the extent of examinees' computer 
experience. First, variety of computer experience was measured by examinee response to the question "For what 
kinds of activities do you use a personal computer?". For this analysis, the variety of types of activities with 
which an examinee had experience was the point of interest; responses were therefore dichotomized as single 
type versus multiple types of activities. Responses to this question confirmed the high level of computer 
experience in this sample. Over 93% of examinees indicated that they used computers for wordprocessing and 
fully 72% had experience with two or more types of software packages. 

Frequency of computer use was measured by the question "How often do you use a personal 
computer?"; frequency of mouse use was measured by the question "How often have you used a mouse on a 
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personal computer?". Responses to both these questions were on a 4-point scale ranging from "l=Never M to 
"4=Routine Use." 

For the Method 1 approach, data were subjected to a three-factor MANOVA; no significant differences 
in scores were found between levels of the computer experience variables examined (see Table 13). 

Table 13 

MANOVA of Residual Difference Scores on Verbal, Quantitative, and Analytical Scales by Variety of Computer 
Experience, Frequency of Computer Use, and Frequency of Mouse Use 



Source 


df 


Wilks' 
Lambda 


F 


Variety of Computer Experience (V) 


3, 1078 


1.00 


.37 


Frequency of Computer Use (A) 


9, 2624 


.99 


.85 


Frequency of Mouse Use (M) 


9, 2624 


.99 


1.27 


V * A 


6, 2156 


1.00 


.43 


V * M 


9, 2624 


1.00 


.19 


A * M 


18, 3050 


.99 


.84 


V * A * M 


12, 2852 


.99 


.81 



Under Method 2, data were subjected to individual chi-square tests (Table 14). Only one significant 
result was found. On the Analytical scale there was an association between mode effect and frequency of mouse 
use, although the strength of the relationship was modest. 

An examination of the related two-way frequency distribution for the Analytical scale included in Table 
15 suggests that the source of the significant chi-squjire can be found in dependence between mode effect 
grouping and the "never" response to the survey question regarding frequency of mouse use. Those examinees 
who had never used a mouse prior to the examination were more likely to fall into the paper mode effect group, 
while those with regular or routine mouse use were more likely to fall into 'he computer mode effect group. 
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Table 14 



Clu-Square Tests of V;iriety ;ind Frequency of Computer Use by Mode 


Effect Group 


Cramer's V 


Tesi 


df Chi-Square 


Variety of Computer Experience 








Verbid 


2 


3.38 


.06 


Quantitative 


2 


0.89 


.03 


Analytical 


2 


4.57 


.06 


Frequency of Computer Use 








Verbal 


6 


5.59 


.05 


Quantitative 


6 


5.05 


.05 


Analytical 


6 


7.66 


.06 


Frequency of Mouse Use 








Verbid 


6 


7.60 


.06 


Quantitative 


6 


11.04 


.07 


Analyticid 


6 


19.90* 


.10 



* £< .01. 
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Tabic 15 

Two-Way Frequency Distribution of Frequency of Mouse Use on Mode Effect Group by Scale 



Frequency of Mouse U se 



Mode Effect 


Never 


Rarely 


Regular 


Routine No Response 








Verbal Scale 






Computer 


25 


46 


53 


34 


1 




(16) 


(29) 


(33) 




t 1 > 


No 


114 


289 


229 


151 


5 


Effect 


(14) 


(37) 


(29) 


(19) 


(1) 


Paper 


31 


53 


42 


39 


2 




(19) 


(32) 


(25) 


(23) 


(1) 








Quantitative Scale 






Computer 


14 


32 


32 


26 


1 




(13) 


(30) 


(30) 




\ 1 ) 


No 


139 


310 


264 


188 


6 


Effect 


(15) 


(34) 


(29) 


(21) 


(1) 


Paper 


17 


46 


28 


10 


1 




(17) 


(45) 


(27) 


(10) 


(1) 








Analytical Scale 






Computer 


7 


29 


40 


16 


2 




(7) 


(31) 


(43) 


(17) 


(2) 


No 


137 


325 


254 


193 


6 


Effect 


(15) 


(36) 


(28) 


(21) 


(1) 


Paper 


26 


34 


30 


15 


0 




(25) 


(32) 


(29) 


(14) 


(0) 



Note. Percentages are given in parentheses. N = 1 1 14. 
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Test Mode Preference 

The next analysis considers in additional computer use variable: test mode preference. Test mode 
preference was measured through responses to the question "If there were a computer test and a paper^and-pcncil 
test with the same questions, which would you prefer to take?". Overall, 61% of the respondents indicated that 
they would prefer the computer mode, while 29% indicated the paper mode, and 1 \ % reported no preference. 

The relationship between mode effect and examinees' test mode preference was juialy/ed under the 
Method 1 approach using a one-way MANOVA of examinees' residualized difference scores on the three 
examination sades by test mode preference. No significant main effect was found for test mode preference, F 
(6, 2150) = 1.80. £> .05, Wilks* Lambda = .99. 

Data were analzyed using the Method 2 approach through individual chksquare analyses. Significant 
icst mode preference effects were found for both the Verbid find Analytical scales of the examination, with x : (4, 
N = 1082) = 10.95, £ < .05 and, xH N = 1082) = 14.94, £ < .01, respectively. No significant effect was found 
for the Quantitative scale, x 2 (4, N = 1082) = 6.06, £ > .05. 

Visual examination of die two-way frequency distributions of mode effect groups by test mode 
preference (Table 16) suggested that on both die Verbal and Analytical scales examinees tended to prefer the test 
mode in which they performed better. 
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Table 16 

Two-Way Frequency Distribution of Tcsi Mode Preference and Mode Effect on Verbal, Quantitative, and 
Analytical Residual Difference Scores 



Test Mode Preference 



Mode 


Computer 


No 


Paper 


No 


Effect 




Preference 




Response 








Verb;d Scale 






Computer 


1 1 0 

1 12 




13 


33 


1 




(70) 




(8) 


(21) 


(1) 


No 


441 




89 


233 


25 


bttect 


(56) 




(11) 


(30) 


(3) 


Paper 


103 




61 


45 


6 




(61) 




(8) 


(27) 


(4) 








Quantitative Scale 






Computer 


68 




9 


20 


8 








(8) 


(19) 


(8) 


No 


522 




97 


266 


22 


Effect 


(58) 




(11) 


(29) 


(2) 


Paper 


66 




9 


25 


2 




(65) 




(9) 


(24) 


(2) 








Analytical Scale 






Computer 


68 




77 


14 


5 




(72) 




(8) 


(15) 


(5) 


No 


523 




103 


263 


26 


Effect 


(57) 




(11) 


(29) 


(3) 


Paper 


65 




5 


34 


1 




(62) 




(5) 


(32) 


(1) 



Note . Percentages are given in parentheses. N = 1 1 14. 



2 V 



25 

Test Strategy Preference 

The next analysis considered the relationship between mode effect and test strategy preference. Test 
strategy preference was measured as examinee reaction to testing constraints on the fimd section of the computer 
version of the examination. Examinees were asked the survey question: "You were not permitted to use the 
'Review' option during the last (seventh) section. What was your reaction to ihis testing rule?". Only 20% of 
the examinees who responded to this question indicated "did not care", while 41% indicated that they found these 
restrictions "somewhat frustrating" and 39% found the constraints "very frustrating". 

The Method 1 procedure for investigating this question utilized a one-way MANOVA of residualized 
difference scores on the examination scales by test strategy preference. A significant main effect for test strategy 
preference was found, F (6, 2040) = 3.01, £ < .01, Wilks' Lambda = .98. Follow-up one-way ANOVAs 
revealed significant effects for the Verbal and Analytical sc;des, F (2, 1023) = 3.44, £ < .05, and F (2, 1023) = 
5.55, £ < .01, respectively. No significant effect was found for the Quantitative scale, F (2, 1023) = 2.30, £ > 
.05. For both the Verbal and Analytical scales, Tukey's post hoc test revealed significant pairwise differences (£ 
< .05) between examinees who responded "did not care" and examinees in the two other groups. The lower 
mean residual difference score for examinees in this group suggests that they performed less well on the 
computer version thutn their paper version scores would have predicted, as compared to examinees in the two 
"frustrated" groups. No significant differences were found between examinees in the two groups "somewhat 
frustrated" and "very frustrated" on either the Verbid or Analytical scales. 

Individual chi-square tests were employed for the Method 2 approach. A significant result was found 
for the Analytical scale, yj{4 y N = 1027) = 14.76, £ < .01. An examination of the two-way frequency 
distribution associated with this test (see Table 17) revealed that twice as many examinees who responded "did 
not care" came from the paper mode group as from the computer mode group. No significant results were found 
for the Verbal or Quantitative sc;des, x 2 (4, N = 1027) = 3.52, £ > .05 and r(4, N = 1027) = 7.86, £ > .05 
respectively. 
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Table 17 

Two-Way Frequency Distribution of Test Strategy Preference and Mode Effect on Residual Difference Scores 

Test Strategy Preference 



Mode Did Not Somewhat Very No 



Effect 


Gire 


Frustrating Frustrating 


Response 






Verbal Scale 






Computer 


24 


0 / 




9 




(15) 


(42) 


(37) 


(6) 


No 


148 


295 


282 


63 


Effect 


(19) 


(37) 


(36) 


(8) 


Paper 


37 


61 


54 


15 




(22) 


(37) 


(32) 


(9) 






vUtUiiiuiuve vjcaie 






Computer 


16 


45 


36 


8 




(15) 






(8) 


No 


164 


341 


329 


73 


Effect 


(18) 


(38) 


(36) 


(8) 


Paper 


29 


37 


30 


6 




(29) 


(36) 


(29) 


(6) 






Analytical Scale 






Computer 


15 


38 


37 


4 




(16) 


(41) 


(39) 


(4) 


No 


160 


354 


326 


75 


Effect 


(17) 


(39) 


(36) 


(8) 


Paper 


34 


31 


32 


8 




(32) 


(30) 


(30) 


(8) 



Note . Percentages are given in parentheses. N = 1 1 14. 
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Test Flexibility 

The final ;trea of analysis in this study was the relationship between examinees' test taking strategy 
(operationally defined as the tendency to omit or review items) and flexibility. Data for this question were 
obtained entirely from the computer Verbal sections of the examination, since flexibility was not manipulated 
elsewhere. In the first six sections of the examination subjects were able to omit and revise items. The seventh 
section of the test was non-flexible, or constrained; examinees could not omit items, and they were not permitted 
to review or revise an item once that item was completed. This non-flexible section was a Verbal section and 
was otherwise completely parallel to the other two Verbal sections. For the non-flexible condition, the dependent 
variable was the simple number correct score. For the flexible condition, the mean number correct score of the 
two flexible Verbal sections was the dependent variable. Means and standard deviations for these scores are 
given by mode effect group and flexibility in Table 18. 

Table 18 

Means and Standard Deviations of Verbal Number Correct Score by Mode Effect Group and Level of Flexibility 



Level of Flexibility 



Mode Effect 



Flexible 



Non-Flexible 



Computer Mode Effect 
M 
SD 



28.26 
4.76 



30.81 
4.73 



No Mode Effect 
M 
SD 



25.83 
5.78 



29.65 
5.91 



Paper Mode Effect 
M 
SD 



19.92 
4.88 



26.45 
5.56 



Note. N = 1114. 
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Data were analyzed through two repeated measures ANO VAs. First was a one-between, one-within subjects 
ANOVA with mode effect group as the betwecn-suhjecLs factor and level of flexibility as the within-subjects 
(repeated measures) factor. A significant main effect for flexibility was found, as was a significant interaction 
between flexibility and mode effect (Table 19). 



Table 19 



Source 


df 


MS 


F 


Between Ss 








Mode Effect (M) 


2 


3747.37 


65.26* 


Error (S/M) 


mi 


57.42 




Within Ss 








Flexibility (F) 


i 


6281.44 


1135.99* 


MF 


2 


356.89 


64.54* 


Error (SF/M) 


1111 


5.53 




Total 


2227 







£<.()!. 



As a follow-up to the interaction effects, correlated means t-tests were computed on the difference between 
the non-flexible and the flexible conditions for each level of mode effect group. Contrary to anticipated results, the 
mean examinee performance was significantly higher under the non-flexible condition than under the flexible 
condition for all three levels of mode effect: for the computer mode effect group, t (159) = 10.99, £> < -01; for the 
no mode effect group, t (788) = 33.19, £ < .01; and for the paper mode effect group, t (167) = 22.31, £ < .01. 

In order to determine whether those examinees whose test taking strategy includes more omissions and 
revisions would be more greatly affected by constrained conditions, a three-between, one-within subjects ANOVA 
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was conducted. The between-subjects factors of this ANOVA, in addition to mode effect, included categorical 
measures of item omits and item reviews; level of flexibility was again the within-subjects factor. 

The measure of item reviews was based on responses to the question: "For how many questions did you 
use the 'Review' screen?" Response options to this question were on a 6-point scale, ranging from " l=None" to 
"6=Almost all." These six categories were collapsed to three (None, A few - 1/4, and 1/2 to Almost All). The 
measure of item omits was obtained by dichotomizing the actuiil count of Verbal items omitted by examinees. 
Examinees were then categorized as Low (0 - 2) or High (3 or more) in omitting. 

The addition of these measures did not add a great deal to the analysis. The behaviors as measured here 
did not appear to be related to mode effect, its evidenced by the lack of interaction between mode effect and either 
item omits or item reviews (see Table 20). The only significant two-way interaction, flexibility by omits, revejiled 
a significant improvement in scores for examinees who were high in the tendency to omit, when the opportunity to 
omit items was not available. 



32 



30 

Table 20 

Thrcc-Bctwccn. One-Within Subjects ANOVA of Verbal Numher Correct Score on Mode Effect Group, Omitting, 
Reviewing, and Flexibility 

Source df MS F 



Between Ss 

Mode Effect (M) 2 266.03 4.77** 

Omit^(O) 1 239.20 4.29* 

Reviews (R) 2 471.96 8.46** 

M* 0 2 60.92 1.09 

M * R 4 34.53 0.62 

O * R 1 2.14 0.04 

M*0*R 2 9.57 0.17 

Error S/MOR 1031 55.81 
Within Ss 

Flexibility (F) 1 364.67 66.49** 

F * M 2 36.62 6.68** 

F* 0 1 22.90 4.18* 

F * M * O 2 2.87 0.59 

F * R 2 11.75 0.12 

F * M * R 4 6.27 0.33 

F * O * R 1 17.78 0.07 

F*M*0*R 2 8.99 0.19 

Error (F/MOR) 1031 548 

Total 2091 



£< .05. ** £< .01. 
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Discussion 

The data in this study demonstrated mode effect and supported the conception of a small subset of 
examinees whose pcrfonmmcc was more affected by test mode than was that of the total sample of examinees (see, 
for example. Table 2). As always, the results should be interpreted in light of limitations on the resejirch design. 
Limitations in this research include the confounding of test mode with order: all examinees took the paper version 
of the examination first and the computer version last. Flexibility iilso is confounded with order. Additionally, 
certain of the variables were measured less precisely than would be optimal. As an example of better measures of 
variables, the test taking strategy of item reviews could be more appropriately obtained in future research through 
test software in which an actual count is taken of the number of times an examinee views each item. 

The search for examinee characteristics that could explain the occurrence of mode effect yielded inconsistent 
results. The variables investigated in this study showed only relatively weak relationships with mode effect. Results 
under the Method 1 and the Method 2 approaches were kirgely parallel, as the summary of results in Table 21 shows. 
It is possible that more appropriate variables exist, or that measurement of the variables investigated in this study 
could be improved; either of these cases could yield evidence of stronger relationships between examinee 
characteristics and mode effect. 
Demographic Variables 

An overall MANOVA on the demographic variables of gender, racial/ethnic background, and age showed 
significant main effects for gender jmd race, and a significant race by age interaction (Method 1). However, these 
results varied across the three scales of the GRE General Test and some of them failed to hold up under pairwise, 
post hoc analyses. 

Significant effects were found for gender on both the Verbid and Quantitative scales. Interestingly, on the 
Verbal scale, the gender by race interaction effect showed the mean residual score for non-White males io be 
negative, while for non-White females it was positive. This indicates that, based on their perfonnance on the paper 
version of the Verbal scale, males on average performed less well on the computer version of that scale than would 
be expected. On the Quantitative scale the pattern of performance was reversed; a main effect for gender was found 
in which males had higher mean residual difference scores than females. That is, the performance of males on the 
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Tabic 21 

Summary of Results for Statistical Tests of Significance Under Method 1 and Method 2 by Research Question 



Statistical Results 



Method 1 



Method 2 



Demographics 



MANOVA 

ANOVA - Verbid 
Quantitative 
Analytical 



MANOVA 
Verbal 
Quantitative 
Analytical 



sig 
sig 
sig 
sig 



ns 



Log-linear 
Verbid 
Quantitative 
Analytical 

Computer Experience 

Chi-Square 
Verbid 
Quantitative 
Analytical 



ns 
sig 
sig 



ns 
ns 
sig 



Test Mode Preference 



MANOVA 
Verbal 
Quantitative 
Analytical 



ns 



Chi-Square 
Verbid 
Quantitative 
Analytical 



sig 
ns 
sig 



Test Strategy Preference 



MANOVA 

ANOVA 
Verbal 
Quantitative 
Analytical 



Main Effect for 

Flexibility 
Flexibility x 

Omits 
Flexibility x 

Reviews 



sig 

sig 
ns 
sig 



Chi-Square 
Verbal 
Quantitative 
Analytical 



Tendency to Omit/Review (Verbid) 

sig Flexibility x 

Mode Effect 
sig Flexibility x 

Omits x Mode Effect 
ns Flexibility x 

Reviews x Mode Effect 



ns 
ns 
sig 



sig 



ns 



ns 
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computer version of the Quantitative scale was better than their paper Quantitative scores would predict, while 
females performed less well on the computer Quantitative scale than their paper Quantitative scores would predict. 

On both the Verbal and Quantitative scales, a race by age interaction revealed a significant difference 
between the residual difference scores of older and younger Black examinees. Older Black examinees performed 
less well on the computer version than their paper version scores predicted, as compared to younger Black 
examinees. On the Analytical scale, a main effect for race, followed by pairwise comparisons between races, 
revealed a significant difference between Asians and Blacks. The difference between residual scores for Asians and 
Blacks suggested that, based on their paper Quantitative scores, Asians did better on the computer Quantitative scale 
as compared io Blacks. 

Results of the log-linear analysis (Method 2) revealed main effects for both gender and age in the 
Quantitative scale and a main effect for age in the Analytical scale. Follow-up examinations of bivariate frequency 
distributions for the Quantitative scale suggested that females were found in the paper mode effect group and no 
mode effect group to a greater extent than in the computer mode effect group. Younger examinees were found in 
gicater frequencies in the no mode effect group and the computer mode effect group, relative to the paper mode 
effect group for both the Quantitative and Analytical scales. 

Interpretation of the mixed results found for demographic variables is difficult. It may be that mode effect 
is not directly related to any of the demographic variables investigated. Perhaps, instead, mode effect is an indirect 
outcome of some other variable, such as test anxiety, and its interaction with certain demographic attributes. Such 
an interaction might explain the differing patterns of performance by males and females on the Verbal and 
Quantitative scales. 
Computer Use Variables 

Minimal results were found in this study to support the relationship between computer experience and mode 
effect. When the data were subjected to a three-factor MANOVA (the Method 1 approach), no significant 
differences in residual difference scores were found between levels of the computer experienc * variables examined. 
Only one significant result was found for the Method 2 approach of individual chi-square tests: on the Analytical 
scale there was an association between mode effect and frequency of mouse use. Examination of the two-way 
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frequency distribution of these variables suggested that examinees with no prior mouse use were over-represented 
in the paper mode effect group, while those with regukir or routine mouse use were more strongly represented in 
the computer mode effect group. 

Similarly, few results were found for the relationship between test mode preference and mode effect. A 
MANOVA. of examinees' residualized difference scores on the three examination scales by test mode preference, 
the Method 1 approach, reveled no significant effect. The individual chi-square analyses conducted as the Method 
2 approach revealed significant test mode preference effects for both the Verbid and Analytical scales. For both 
scales, the relationship appeared to be a tendency for examinees to prefer the test mode in which they performed 
better. 

Perhaps it was because of the high level of computer experience in this sample that investigation of 
computer use variables failed to show any strong results. The high representation of computer experience in this 
sample was somewhat expected, given that participation in the computer test pilot study was entirely voluntary. 
Nevertheless, the lack of subjects with no computer experience makes analysis of the relationship between mode 
effect and computer use variables very difficult. 

Lee (1986) found that while examinees with no computer experience were negatively affected by computer 
administration of a test, those examinees categorized as "low" on computer experience were not. It may be that as 
Wainer, Dorans, Green, Mislevy, Steinberg, and Thissen (1990) have suggested, computer tests require so few 
computer skills that frequency and ariety of computer experience do not impact mode effect. On the other hand, 
the extent of computer experience which is beneficial to an examinee may vary across different computer test 
conditions; the impact of computer experience on mode effect may depend upon the complexity of the test 
administration software. Test administration software with a cumbersome interface has been noted for increased 
testing time and increased examinee anxiety (Karvey, 1987; Johnson & Johnson, 1981). Differences in testing 
software could confound the relationship of computer experience to mode effect. 
Test Taking Strategy Preferences 

Examinee test taking strategy preferences w;is considered to be an important variable for analysis in this 
study because the two test modes of computer and paper versions may match up differentially with different strategy 
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approaches. If examinees strongly prefer one set of strategies, and the test mode processes conform well to those 
strategies, then those examinees may be aided in their overall performance under that lest mode. Conversely, if a 
given lest mode restricts the use of those strategies but provides a good match for other strategies, then those 
examinees who are able to adjust to these processes and apply new strategies will be better off than those examinees 
who ;ire not able to do so. 

This study was specifically concerned with test taking strategy as it applies to differences in levels of 
flexibility across test modes. When flexibility is discussed as an issue in computer testing it is often from the 
perspective that not blowing examinees to omit and revise items may penalize them, compared to their potential 
performance on the same test in paper form (Spray el al., 19X9; Ward et al,, 1989), In addition, the literature on 
answer changing on paper tests consistently shows that examinees are likely to improve their scores when they 
change their responses to test items (Green, 1981; Matter, 1986: Mueller & Shwedel, 1975; Penfield & Mercer, 
1980), This would tend to support the need for providing flexibility in computer tests. The results of this study, 
however, would at first glance appear to support the opposite position; the overall mean score for the non-flexible 
Verbal section of the examination was higher than for the flexible sections. Contniry to results found in other studies 
on flexibility (Spray et al., 1989; Ward et al., 1989), performance was significantly higher in the non-flexible 
condition; although this was true at all three levels of mode effect, the difference was especially marked in the paper 
mode effect group. 

Despite the overall performance increase in the non-flexible section, the non-flexible condition was a source 
of frustration to most examinees. A total of 80% of the examinees indicated that they found the constraints on 
omitting and revising to be frustrating. The relationship between test strategy preferences and mode effect was 
examined through Method 1 ANOVAs and significant effects were found for the Verbal and Analytical scales. 
Those examinees who indicated "did not care" had a significantly lower mean residual score, indicating that their 
computer test performance was not as high as their paper test performance predicted, as compared to examinees who 
selected "somewhat frustrated" or "very frustrated". Under Method 2, a significant chi-square test was also found 
for the Analytical scale. Examination of the bivariate frequency distribution between test mode preference and mode 
effect groups also suggested that the "did not care" response was associated with the paper mode effect group. One 

38 



36 

possible interpretation of these results is that those exiiminees most comfortable with the sequential pattern of 
responding to items forced by non-flexible conditions would also be those whose performance would be higher in 
typical paper administrations. 

Test taking strategy and test flexibility may be related in more complex ways than have yet been fully 
studied. Issues concerning speededness, the accessibility of (he test administration software, and the structure and 
orgsinization of the exwnination may all contribute to delineating the best strategy for a given test administration. 
More precise measures of examinees' item omitting and reviewing behaviors ought to be collected to further 
investigate the relationship of these strategy preferences with mode effect under the constrained conditions often 
found in computer testing. 

Summary 

The presence of a mode effect was delected in the data used in this study. Specifically, while the majority 
of examinees were not affected by tesi mode, one subset of examinees performed better on the computer version, 
and a second subset of examinees performed better on the paper version of the test. However, the se;irch for 
examinee characteristics that explain the occurrence of mode effect yielded mixed results. The variables investigated 
in this study showed only relatively weak relationships to mode effect. Further investigation tailored to this question 
should be conducted in order to determine those variables which distinguish those examinees whose performwice 
is affected by mode of test administration from those whose performzince is not. 
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